The present invention relates to digital imaging systems. More particularly, the present invention relates to the technique of detecting and compensating for illuminant intensity changes in digital imaging systems.
In digital cameras, a scene is captured by using a lens to form an image of the scene on the surface of an array of sensors, such as photodiodes. Each sensor detects light from a tiny portion of the scene. At each sensor, the detected light is converted into an electrical signal, and then into a digital value indicating the intensity of the light detected by that sensor. Then, the digital values from all of the sensors of the array are combined to form an image.
Popular sensor arrays include CMOS (complementary metal-oxide semiconductor) sensors and CCDs (charge-coupled devices). The sensor array often includes a rectangular layout of many hundreds of thousands, millions, or even greater number of sensors, each sensor providing a digital value, or a pixel, of information. For example, a rectangular sensor array arranged in 640 columns and 480 rows has 307,200 sensors, or pixels. A digital value from a sensor is defined as a pixel of the image. For convenience, terms “sensor” and “pixel” are herein used interchangeably unless otherwise noted, and each sensor, or pixel, is referred generically as Pi,j where i,j indicates that the pixel is located at ith column at jth row of a rectangular sensor array having M columns and N rows, the value of i ranging from 1 to M, inclusive, and the value of j ranging from 1 to N, inclusive.
To capture the scene, the electrical value of each of the sensors of the array is read serially. That is, the camera reads the digital values of each of the sensors beginning with the first sensor, P1,1 and ending with the last sensor, PM,N. Typically, the pixels are read row-wise, that is, row by row. To read each sensor, the sensor is first reset to a predetermined value, for example, zero. Then, the value of the sensor is read after an exposure period. The exposure period determines how long the sensor being read has been exposed to the scene. The exposure period is also referred to an integration period. This is because the sensor sums up, or integrates, the light it receives during the exposure period. A rolling shutter technique is often used to read the sensor array. The rolling shutter is implemented by sequentially resetting each row of sensors and sequentially reading the values of each row of sensors. The duration or the period of time between the reset and the read is the integration period. The period of time taken to read the entire array of sensors is often referred to as a frame period or an image capture period.
The scene captured during the frame period is often illuminated by an electrically powered light having flicker. Flicker is the wavering of the characteristics of light source with time, including variations of light intensity, color temperature, or spatial position. Flicker is often too rapid for the human eye to detect. Some common light sources that exhibit flicker are fluorescent lights often used in office and industrial settings and tungsten halogen incandescent lamps. For example, in fluorescent lamps, the phosphors in the lamp are excited at each peak in the waveform of the AC (alternating current) power source. Between the peaks of the AC power, the light intensity is diminished. The light pulses that are produced have a flicker frequency that is twice that of the AC source. In the United States of America, the commonly available AC power has a sinusoidal waveform of approximately 60 Hz producing light having 120 Hz flicker frequency. That is, a fluorescent lamp produces light that cycles from high intensity to low intensity at approximately every 120th of a second. Thus, the flicker period is 8.3 milliseconds (ms). The flicker period can vary from country to country depending on the frequency of the AC power source. The flicker due to the AC power waveform is illustrated in FIG. 1 as illuminant intensity curve 12 of the graph 10.
If the light flickers during the frame period, then undesirable artifacts can appear in the captured image. This is because some portions, or rows, of the sensor array are exposed to the scene and read when the scene is illuminated with relatively high intensity light while other portions, or rows, of the sensor array are exposed to the scene and read when the scene is illuminated with relatively low intensity light. Such undesirable artifacts can include, for example, variations in brightness and horizontal bands within a captured image.
A common frame rate of various sensor arrays is 30 frames per second. That is, typically, it takes about 33.3 ms to capture a scene. The frame period is also illustrated in FIG. 1 as beginning at time T0 and ending at time T8. A frame period can include one or more flicker periods producing visible artifacts in the captured image. As illustrated, within the frame period, four flicker periods occur, each flicker period having a period of time of relative low intensity of the light from the illuminant.
FIG. 2 illustrates a sensor array 20 in a rectangular grid having M columns and N rows of pixels. For example, M can be 640 and N can be 480. The first pixel is illustrated as P1,1, a generic pixel at Column i and Row j as Pi,j, and the last pixel at Column M and Row N as PM,N. To avoid clutter, not all rows or columns are illustrated; however, the presence of the columns and the rows not illustrated are indicated by ellipses 22. FIG. 2 also illustrates the graph 10 including the illuminant intensity curve 12, rotated.
To capture a scene lighted by an illuminant, the sensor array 20 is read beginning at time T0, row-wise, and ending at time T8. On one hand, at times T0, T2, T4, T6, and T8, light from the illuminant is at its highest intensity. On the other hand, at times T1, T3, T5, and T7, light from the illuminant is at its lowest intensity. Accordingly, rows of sensors detecting light at or near these times (T1, T3, T5, and T7) receive relatively less light than the rows of sensors detecting light during the other time periods (T0, T2, T4, T6, and T8). For convenience, the rows of sensors detecting light at or near the times T1, T3, T5, and T7 are referred herein as ROW(T1), ROW(T3), ROW(T5), and ROW (T7), respectively. This results in a final image having dark bands at these rows.
To alleviate this problem various methods have been suggested. For example, one approach that is suggested involves the use of histograms of light levels of different image frames. The histograms are compared so that the variation in the mean level of illumination as a function of time can be removed. This approach is not applicable where only a single image frame is available. In another approach, the integration period is restricted to integer multiples of the flicker period. This places an undesirable lower limit on the integration period and introduces possible overexposure issues for brightly illuminated scenes. Other approaches include use of mirrors, prisms, and other bulky and expensive components to marginally alleviate the artifacts problem with varying degree of success.
There remains a need for an improved digital imaging system that detects illuminant intensity changes within an image capture period and compensates the captured digital image from effects of the illuminant intensity changes.